Repository navigation
fix(cluster): end when quit() or disconnect() is called while reconnecting - #2240
Merged
PavelPashov merged 2 commits intoOct 21, 2026
Merged
PavelPashov merged 2 commits into
PavelPashov merged 2 commits into
Conversation
…cting When every node connection is lost, the cluster goes to "reconnecting" with an empty connection pool. quit() and disconnect() there set "disconnecting" and wait for the pool's "drain", which an empty pool never emits, so the cluster never reaches "end" and later commands stay queued. They now take the branch "wait" already uses. On a cluster that is already "end", both keep the status instead of moving it to "disconnecting" for good.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Keep the after-"end" status change out of this PR, and trim the tests to one per code path: disconnect() and quit() while reconnecting, and disconnect(true) still waiting for its pending reconnect.
PavelPashov
approved these changes
Oct 21, 2026
PavelPashov
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thank you for your contribution
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When every node connection drops, a
Clustergoes toreconnectingwith an empty pool. If you callquit()ordisconnect()at that point, it stays indisconnectingfor good:endnever fires and anything sent afterwards sits in the offline queue.Cause: the pool's
drainhandler (L255-L260) setsclose, the one-shotcloselisteners fromconnect()(L393-L394) fire, andhandleCloseEvent()(L1099) setsreconnectingand arms the retry timer. The pool is empty by then.disconnect()(L425-L451) andquit()(L456-L500) setdisconnectingand clear the timer, but they only move on by hand fromwait. Every other status waits for the pool'sdrain. Andreset([])on an empty pool emits nothing, with nocloselistener left, so nothing ever ends the client.sendCommandqueues whiledisconnecting, so later commands just hang.Repro on a real Redis 8.6 cluster (3 masters + 3 replicas): kill this client's connections on all 6 nodes with
CLIENT KILL, then callquit()ordisconnect()on thereconnectingevent, withclusterRetryStrategy: () => 1500. On main all 6 runs (3 of each) hung:Fix: both methods take the existing
waitbranch forreconnectingtoo, so they runsetStatus("close")+handleCloseEvent(). WithmanuallyClosingset, that setsendand flushes the offline queue with "None of startup nodes is available".quit()keeps itssetImmediateand still resolves"OK". 3 insertions, 2 deletions inlib/cluster/index.ts.I left
disconnect(true)alone. Its timer isn't cleared and still callsconnect(), so runninghandleCloseEvent()on top would callclusterRetryStrategya second time and arm a second timer.Same cluster with the fix, 6 of 6 end:
handleAbortedConnect()from #2217 (L1081-L1094) handles the same "disconnecting with an empty pool" case, only inside aconnect()aborted while resolving the startup nodes. This uses the sameclose+handleCloseEvent()pair.Behaviour change: a cluster you
quit()ordisconnect()while it's reconnecting now emitsend, and its queued commands get rejected. I can't think of code that depends on the hang, but it is a visible change.Tests:
3 new tests in
test/functional/cluster/disconnection.ts, withMockServer, next to the existing one for nodes ending afterdisconnect(). Withlib/cluster/index.tsat main, thedisconnect()andquit()ones fail:The third, "should not retry again when disconnect(true) is called while reconnecting", passes on main and on the branch. It pins that
disconnect(true)still waits for its pending reconnect (clusterRetryStrategycalled once).npm run test:js(Redis 8.10.2, Node 22): 2908 passing, 25 pending, 1 failing. The failure isautoPipelining for single node"should handle large pipelines without RangeError", an 8000 ms timeout that also fails on main now and then and passes alone.npm run test:clusteragainst a real 6-node cluster: 71 passing.npm run buildpasses.npm run lint: 0 errors, the 23 warnings already on main.prettier --checkclean on both files.Note
Medium Risk
Changes cluster teardown semantics during reconnecting; callers that relied on staying in disconnecting will now get end and rejected commands, but this fixes a stuck-state bug in connection lifecycle handling.
Overview
Fixes a hang when
quit()ordisconnect()is called while the cluster is inreconnectingwith an empty connection pool (no node left to emit pooldrain). Those paths now mirror the existingwaithandling: they drivecloseandhandleCloseEvent()directly instead ofconnectionPool.reset([]), soendfires,manuallyClosingprevents another retry, and queued commands are rejected.disconnect(true)is unchanged—it still does not take this shortcut, so a single pending reconnect fromclusterRetryStrategyis not double-scheduled.Adds functional tests for
disconnect/quitduringreconnectingand assertsdisconnect(true)does not invoke the retry strategy again.Reviewed by Cursor Bugbot for commit 1866bf1. Bugbot is set up for automated code reviews on this repo. Configure here.